Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bugfix/parameterised_thread_count #158

Merged
merged 1 commit into from
Nov 1, 2023

Conversation

thomas-fred
Copy link
Contributor

to dynamically set the number of threads (and amount of memory) for certain rules (create_power_network and estimate_wind_fields) we use a dynamically generated CSV file with the number of targets in each country

this file must be present for rules to allocate resources

use the checkpoint lookup in an input function to guarantee its presence

previously, we had removed the checkpoint lookup from the threads_for_country function as per the docs, but this does not seem to work (snakemake error on comparison of thread count integer and 'TBDString')

maybe this whole scaling resources is too much complexity, but trying to run a global analysis without it is also inefficient at the aforementioned rules...

to dynamically set the number of threads (and amount of memory) for
certain rules (create_power_network and estimate_wind_fields) we use a
dynamically generated CSV file with the number of targets in each
country

this file must be present for rules to allocate resources

use the checkpoint lookup in an input function to guarantee its presence

previously, we had removed the checkpoint lookup from the
`threads_for_country` function as per the docs, but this does not seem
to work (snakemake error on comparison of thread count integer and
'TBDString')

maybe this whole scaling resources is too much complexity, but trying to
run a global analysis without it is also inefficient at the
aforementioned rules...
@thomas-fred thomas-fred merged commit ab9e7a5 into main Nov 1, 2023
@thomas-fred thomas-fred deleted the bugfix/parameterised_thread_count branch November 1, 2023 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant